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Abstract — This method is based upon the image registration 
process and the application is when the text which is to be 
identified is behind the mesh which works as a hurdle. We 
know that the mesh as hurdle can be made less irritating by 
either moving the camera or the source itself. The method 
uses Radon Transform for extracting the mesh lines and 
capturing the position of the mesh lines. The final process of 
filling the deformed image is through the registration. The 
method is adaptive to movement in any direction. The 
transformation errors are removed by the Quarterions. It was 
tested on a number of images [200] approximately and gave 
excellent results. 

Index Terms — Image Registration, Image Processing, Image 
in motion, Quarterions. 

I. Introduction 

In computer vision, set of data acquired by sampling the 
same scene or object at different times, or from different per- 
spectives, will be in different coordinate systems. Image reg- 
istration is the process of transforming the different sets of 
data into one coordinate system. Registration is necessary 
in order to be able to compare or integrate the data obtained 
from different measurements. The Multi View Image Regis- 
tration technique uses images of an object taken from differ- 
ent views called modalities. Many interesting and useful ap- 
proaches to the multiview registration problem have been 
proposed in the recent past. Chen and Medioni, and Masuda 
and Yokoya both incrementally register views against a grow- 
ing global union of viewpoints [3], [6] . . . Bergevin et al.place 
all views into a global frame of reference, and then repeatedly 
select a view and register it against all others [2]. Stoddart 
and Hilton find pairwise correspondences between points in 
all views, and then iteratively minimize correspondence er- 
rors over all views using a descent algorithm [1 1]. This basic 
technique is extended using a multiresolution framework, 
surface normal processing, and boundary point processing 
by Neugebauer [7] and Eggert et al. [4]. Williams and 
Bennamoun suggested a further refinement by including in- 
dividual covariance weights for each point [12]. Sawhney et 
al. and Shum and Szeliski perform the global alignment of two 
dimensional mosaics by nonlinear minimization of distance 
between point correspondences [8], [10]. Gregory, Lee, and 
David address the problem of large-scale multiview registra- 
tion of range images captured from unknown viewing direc- 
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tions [9]. 

Benjemaa and Schmitt propose a nonlinear solution based 
on the quaternion representation [1]. Their formulation is a 
multiview extension of the pairwise solution proposed by 
Horn [5] using distance between pairwise correspondences 
as the optimization criterion. Donghun Lee, Jihun Park [15] 
presents a quaternion-based method to estimate the camera 
parameters in a single video stream to interpolate 3D trajectory 
of the moving camera with camera parameters. Jihun Park [16 
] determining camera parameters for three dimensional scene 
reconstruction given an image with three known points on a 
single plane on the image .Robert Bergevin, Marc Soucy, 
Hewe Gagnon, and Denis Laurendeau [ 17] present an 
algorithm that reduces the level of the registration errors 
between all pairs in a set of range views. Chang Yuan, Ge' 
rard Medioni, Jinman Kang, and Isaac Cohen [18] present a 
method for detecting motion regions in video sequences 
observed by a moving camera in the presence of a strong 
parallax due to static 3D structures. The proposed method 
classifies each image pixel into planar background, parallax, 
or motion regions by sequentially applying 2D planar 
homographies, the epipolar constraint, and a novel geometric 
constraint called the "structure consistency constraint." Jung- 
Young Son, Vladimir V. Saveljev, Jai-Soon Kim, Kae-Dal 
Kwack, and Sung-Kyu Kim [19] Proposed a method based 
on 4*4 homogeneous matrices which can predict distortions 
in the perceived image in projection-type multiview 3-D 
imaging systems. Jiangbo Lu, Hua Cai, Jian-Guang Lou, and 
Jiang Li [20] propose a DE technique to accelerate the 
disparity search by employing epipolar geometry. Lou, Hua, 
and Jiang present the system architecture for real-time 
capturing, processing, and interactive delivery of multi-view 
video. 

In this paper, the proposed algorithm is a practical 
approach for solving the global registration parameters using 
the relative motion between view pairs as the error criterion. 
This criterion does not require linearization and, therefore, 
can be used even when the accumulated rotational error is 
large. Furthermore, this criterion does not require point 
correspondences and can therefore be used together with 
robot odometry or any registration method. These different 
modalities are then analyzed and the information obtained is 
fused to form a coherent composition called multimodal image. 

This paper addresses the problem of large scale multiview 
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registration of range images captured from unknown viewing 
directions. The global error distribution method described in 
can be used together with any pair wise registration algorithm 
to perform globally consistent multi view range image regis- 
tration. The paper is concerned with the existence of hin- 
drance in the view of an object, like when a mesh appears in 
between the camera and the object thus making it partially 
unrecognizable. The hindrance in the view or the mesh can be 
considered as a kind of periodic noise and the technique multi 
view image registration has been used for the solution of this 
problem. The application of image registration is useful if and 
only if either mesh, camera or the object is moving relative to 
other two i.e. each frame in unique in terms of information 
with respect to the previous and the next frame. So the three 
possible situations are: 

(i) Mesh and the object are stationary but camera is in 
motion. 

(ii) Camera and the object are stationary but mesh is in 
motion 

(iii) Camera and mesh are stationary but object is in motion. 
Here, the first situation has been considered in which 

mesh and the object are stationary but the camera is moving 
in different directions i.e. at different angles to collect the 
necessary information about the object which is hidden in 
the view at some angle but is accessible at a different angle. 
The movie is then converted into frames. One frame is taken 
as a reference frame and the rest are target frames. Radon 
transform to detect hindrance in the view i.e. mesh in the 
reference frame and the fusion technique, Pixel based fusion 
to fill new information collected from target frames in place 
of hindrance. The method is an extract of the natural 
registration process and hence the approach is proven. The 
paper is arranged as follows: Section-2 gives a briefing of the 
method followed. Section-3 explains the pre-processing, 
which in this case is the separation of the frames and the 
removal of the wired mesh. Section-4 explains feature- 
extraction and the Section-5 explains the registration algorithm. 
The next Section-6 gives the results obtained followed by 
the conclusions in Section-7. 

II. Multiview Registration Process 

Multiview video or free-viewpoint video is an exciting 
application, because it enables users to watch a static or 
dynamic scene from different viewing angles. Generally, to 
provide a smooth multi-perspective viewing experience, 
content producers need to capture a distinct scene with ideal 
quality from multiple camera positions, such as the multiview 
camera setup shown in Fig. 1 , where a multiview image set is 
produced by steadily moving a single video camera around a 
scene of interest along a predefined capture trajectory The 
simultaneous multiple video streams from multiview cameras 
are referred to as multiview video. A multiview video sequence 
can be regarded as a temporal sequence of special-visual- 
effect snapshots, captured from different viewpoints at 
multiple times. Multiview image set can be generated by 
multiple cameras which are positioned at different angles to 
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capture the scene from different angles. Multiview image set 
can also be generated when the camera is stationary whereas 
the target image is moving. Here in this paper we consider 
that a single camera is moving while the target is stationary. 




Mow; e, C-ar.3i 
^ Dueetiai of jr.otL* 

Figure 1: Multiview Detection Scheme 



A. Error due to Transformations 

In Figs. A, the x and y axes are shown as correctly aligned 
and mutually perpendicular. In Figure 2 the xl and yl pair of 
axes are shown as rotated through an angle a relative to the 
x, y pair. The xl, and yl pair are still perpendicular, but as a 
coordinate frame it is not correctly aligned. This is called drift 
or rotational error. 




Figure 2: Rotation of X and Y axes 

In addition to the drift or rotational error, there is a second 
kind of possible distortion of a coordinate frame called scale 
error. Pure scale error exists when the xl and yl axes, 
respectively, coincide with the x and y axes, but the vectors 
which are supposed to be unit vectors fail to have unit length. 

B. Rotational Error 
D= 

(1) 

Let l,m and n be the components of angular velocity as 
functions of time. Then q(t) is found as a solution of the 
system of differential equations 
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Let's denote the square matrix above as W then we can rewrite 



q'=^Wq 



(3) 



Let the transition matrix corresponding this be Z (t, to). Then 
we may represent the solution of (2) as 

q(t) = Z(t,to)qO (4) 
Where qO represents the initial condition quaternion at 
time tO. Now suppose that f represents an additive noise 
perturbation to (4) due to error. Let the corresponding 
perturbed solution be p(t). We will assume that the initial 
value, p(t0), is likewise equal to qO. The perturbed differential 
equation is 

p'=(l/2)Wp + f (5) 
The solution to this equation is 

p (t) = Z(t, t0)q0 + f f f D Z (t Z}f ii) d t (6) 
Denote the integral term in (6) by r (t). Comparing (4) and (6) 
p(t) = q(t) + r(t). " " (7) 

By definition, 

I g I = 1 (8) 
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Also introduce a related matrix T as 
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(B) 



Considering D as an orthogonal direction cosine matrix, 
the transformation matrix in four dimensional spaces can be 
written as 



A 4 



(14) 
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Figure 3: Quarterion Perturbation 

A geometrical analog in two dimensions for the quaternion 
perturbation is shown in Figure 3. The quaternion q is always 
confined to the circumference of a circle of radius 1 . The 
perturbation r may be in any direction with equal likelihood. 
Given only the vector p, without being able to make reference 
to q, the only way we can tell that a perturbation has occurred 
will be from the fact that 

fl q E * l (9) 

Since there is no way we can estimate the angular deviation 
of p relative to q, the only realistic correction we can make is 
to rescale p back to the circumference of the unit circle in 
such a way that the angular deviation is equally likely to be 
positive or negative. Therefore, introduce the corrected 
quaternion p* obtained by 



p*(t) = p(t)/||p(t) 
p (t) tpl2(t) + p22(t) + p32(t) +p42(t)] 1/2 



(10) 
(11) 



Since the true quaternion q is not available to us, use p* 
in place of q in (1) in order to compute D. Now construct a 
4*4 matrix S having the quaternion components as its 
elements 
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These matrices commute. Further by quaternion 
properties we can say 

A=ST = TS (16) 
Now apply the above procedure to an error analysis. By 
(7), the effect of computational errors will be that the matrixes 
S and T are formed from the elements of p rather than q. 
Denote the results as Sp and Tp. Further, denote the matrixes 
formed in the same way using the components of r by Sr and 
Tr. 
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Then considering (7) 



S + S r 

:T + T 



From (15) 



(19) 



(20) 
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S T =(S+S)(T+T) = ST + ST+ST +S T 

p p r y v r y r r r r 

Neglecting the term S T Direct computation will reveal 
that the pair of matrixes S and T, as well as the pair of matrixes 
S and T , actually commute. Rearranging (20) 
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S p T p = ST(I 4 + S'S r +T'T ) (21) 
Here S' and T are the transpose of matrixes S and T. Let 
the perturbed version of A be called B. Then equation (21) 
may be written as 



B. =A. (I. + H.) 

4 4 v 4 V 



Here I is a 4*4 identity matrix and 



H,=S'S +T'T 



(22) 



(23) 



the matrix H4 may be split into its symmetric part V and its 
anti symmetric part U 4 : 



S 4 = ^(H 4 +H' 4 ) 



(24) 



U 4 = (H 4 -H' 4 ) (25) 
The matrix \ 4 represents the scale and skew errors, and 
the matrix U 4 represents the drift errors, which have crept into 
the computation of rotation via quaternion. 

Reconsidering the equation (5). The sign of the 
perturbation vector f could, in fact, may be plus or minus. 

Therefore, the sign of r in (7) could equally likely be plus 
or minus. Similarly the sign of H4 in (23) could equally likely 
be plus or minus. Hence, the algebraic signs associated with 
\4 and U4 maybe plus or minus. Substituting (12), (13) and 
(17), (1 8) into equation (23) and the result then putting in (24) 
one finds that 



S= 2(q.wr 1 +q.xr 2 +q.yr 3 +q.zr 4 ) I 4 



(26) 



Therefore, if the elements of \4 are positive, it means, 
referring back to Fig. B, that the tip of r lies outside the unit 
circle. Similarly, if \4 has negative elements, it means that the 
tip of r lies within the unit circle. At the same time, the 
associated rotational error could equally likely be in either 
direction. 

We can now readily see that our proposed correctional 
procedure, represented by (10), will have the effect of reducing 
VI to zero. Furthermore, while this procedure will not drive U4 
to zero, it will, in fact, almost always reduce the norm of U4. 
At worst, in a few situations (i.e., when IIPII = 1 to begin 
with), the correctional procedure will have no effect on U4. 

III. The Pre-Processing 

The first step is to extract the frames from the video. One 
of these frames is chosen to be the reference frame and the 
rest are target frames. Since the images have been taken from 
a single optical device, they are classified as single -modality 
images. The reference image is processed further so that 
hindrance is removed from the view and the target images are 
registered into the reference image to make the object 
recognizable. Following is the method followed: 
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Algorithm 1: Extraction of frames 



1. Read in the video file 

2. Determine the suitable number of frames required per 
second 

3. Extract the frames from the video clip 

4. Choose an appropriate reference image from these 
frames 
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Figure 4(a): reference frame (original) 
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Figure 4(b): Target frame (original) 

Let the captured views lie within coordinate frames 
Vl,...,Vn, where two frames Vi and Vj are related by the rotation 
and translation (Ri,j, ti,j). A point pj in frame Vj can be 
described in frame Vi according to the change of coordinates 



pi = Ri,jpj + ti,j 



(27) 
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In practice, the change of coordinates between 
neighboring views will be found using odometery or a pair 
wise registration procedure [1]. We shall call such changes 
of coordinates, measurements. 

The next step is to detect and remove the mesh lines. In 
order to determine the sections of the pictures to be fused, 
the exact coordinates of the pixels corresponding to the noise 
in the picture should be known. The noise in the reference 
image is in the form of mesh which is nothing but an array of 
horizontal and vertical lines. Since, the radon transformation 
is a very powerful technique for the detection of straight 
lines; it has been used for the purpose of mesh detection. 
The detection of the mesh lines is achieved with the 
Algorithm-2. In this algorithm 

1 . First the image is read as a three dimensional matrix 
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with each element representing some intensity values 
Corresponding to three colors red, green and blue (RGB 
image). 

2. Before applying radon transform, the image is converted 
into grayscale image which is a two dimensional image with 
intensity values varying from to 255, corresponding to 
black and 255 corresponding to white. 



Algorithm 2: Mesh Detection 



Given: frames 

1. for all available frames, do 

2. Read in the grayscale image IMG 

3. RADOW^-Radon_Transform(IMG) 

4. for ANGLE*- (1 to 3) & {88 to 92) & (1 78 to 180), 
do 

[to consider only the vertical and horizontal mesh 

lines] 

£ ; If RADON[L ANGLE] >= 

Me an (Pe ak_v alu e s) 

6. S0RTED_RADON[LANGLEl*RAD0N(L ANGLE) 

7. end if 

8. end for 
9. 

im' r _RADONfinverse_E.adon_Transform(SOE.TED_RA 

DON) 

10. 

[WID ILL START_X, START_Y]4"Me sh_Stats (ENVJRAD 
ON) 

LL MAPPEDJMG* 
Mappin g(IMG : HOR_GAP : VERT_GAP : 



START_X : S TART_YAVID TH) 
12. End for 



3. Radon transform [14] is applied on the grayscale image to 
detect the lines representing the mesh. Figure-6(a) shows 
the radon transform for figure-5. Figure-6(b) shows the 
sorted out peak values from the Radon transform that 
correspond to the straight of the mesh. 





Figure-5: Randon Transform of Grey scale image 

The Radon transform gives projection of the image 
intensity along a radial line oriented at a specific angle varying 
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form 00 to 1800. Radon transform provides with the radial 
coordinates of the lines corresponding to each projection 
and corresponding intensity values. 

The Radon transform for a set of parameters (rho, theta) 
is the line integral through the image g(x, y), where the line is 
positioned corresponding to the value of (rho, theta). The 
delta is the Dirac delta function which is infinite for argument 
and zero for all other arguments (it integrates to one) [13]. 

Equation (28) and (29) represent the above expressions. 

rtXi rSi 
J -00 J -co 



(28) 



and the identical expression 



gift. 9} = I <i{f>tw9-$s\n9,psi\)8 + $CQ88) d$ 

(29) 

The very strong property of the Radon transform is the 
ability to extract lines (curves in general) from very noisy 
images as shown below in Figure-6. 
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(b) 

Figure-6 : Sorted peak from Randon Transform 

From the Radon transform, shown in Figure-7, it can be 
seen that crossing lines makes no problem. 
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Figure-7: Removal of lines by Randon Transformation 

The peak points of intensity values correspond to straight 
lines in the reference image. A simple thresholding algorithm 
is applied to pick out the line parameters. 

4. Inverse radon transform is applied to the line parameters 
obtained in the previous step which give lines representing 
the mesh. The inverse radon transform inverts the radon 
transform, and can therefore be used to reconstruct images 
from the projection data. It uses the radial coordinates of the 
line and the angles of projection to get the lines in the image. 
Radon uses parallel-beam projections to reconstruct the 
image. In parallel-beam geometry, each projection is formed 
by combining a set of line integrals through an image at a 
specific angle. As the number of projections i.e. length of 
theta increase, the reconstructed image is more accurate so 
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the length of theta vector has been taken from 00 to 1 800 and 
theta must increase monotonically with a constant increment 
of 10. 

5. These lines are further used to map the mesh or the 
noise and find the total area along with the coordinates of 
pixels where the information is to be filled during fusion. 

IV. Feature Extraction 

In order to obtain the new information to be filled following 
parameters should be known. 

1. Quantization of Information In Successive Frames 

The amount of the information changes in successive 
frames should not exceed the width of mesh i.e. should not 
change abruptly, because if it does the embedded information 
will overlap the previous information and the final image will 
be distorted. So the determination of optimized number of 
frames captured per second is very important before they are 
processed. 

2. Direction of Change in the Information 

Since a camera has been used for the movie of the object, 
the camera can be considered as a point that can move 
randomly in all possible directions at any possible angle in 
the space but only in a fixed plane. The direction of movement 
of camera can be classified in two major categories. 

ii. Perfect horizontal or vertical motion . 

iii. The inclined motion. 

The direction vector is calculated by dividing the 
information in between the mesh of the reference image into 
4, 9 or 16 equal parts depending upon the need of accuracy 
or computational time. More is the number of parts, higher is 
the accuracy but more will be the computational time. So, 
need is to find out the optimized value, which comes out to 
be 9. The information from the next frame from the same 
coordinates is calculated and divided into 9 equal parts. Now 
the correlation of each corresponding part is calculated. The 
part which has undergone maximum change gives the lowest 
value for correlation. Thus the minimum value of correlation 
gives an approximate value of direction vector i.e. movement 
of camera. The direction vectors 2, 4, 6 and 8 correspond to 
perfect horizontal and vertical motion and 1, 3, 7 and 9 
designate the inclined motion respectively. (Figure-8 shows 
a single cell from the target frame Figure 4). 




Figure 8:Single cell from the target frame Figure 4 

3. Pixel positions of New Information 

The registration process being implemented is feature- 
based registration so the features changed in the new frame 
with respect to the previous one have to be extracted before 
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the fusion is done. The quantization determines the amount 
of features changed and direction vector determines the 
coordinates of new information i.e. the new information is 
specific for a particular detection vector and extraction of 
features is also divided into two categories: 

i. First is when the information is changing linearly exactly 
horizontally or vertically. The new information is rectangular 
and can be easily detected by merely comparing the new 
frame with the previous one along each column. 

ii. Second condition deals with the condition when the camera 
is moving at an inclination i.e. information enters into the 
frame diagonally. This time new information is not uniform 
and extraction is much more complicated and is drawn out by 
successively comparing the new frame with the older one 
along both rows and columns. 

V. Fusion technique 

The fusion technique being used is pixel based fusion 
and involves placement of information obtained from previous 
step in place of mesh. The coordinates of the mesh in the 
reference frame have already been determined like the 
previous step fusion also depends upon the direction vector 

The new information is embedded into the reference frame 
which gives noise free information i.e. views of object without 
any kind of hindrance. . (Figure-9 shows the resultant image). 

VI. The Registration Algorithm 



The Following Algorithm (Algorithm-3) is the one used 
to implement the multi view registration of the images 




Figure 9:The Resultant Image 

The purpose of the algorithm is to perform the pixel based 
fusion on a pair of frames. One is the reference frame called 
the base frame (BASE_IMG) and other is one of the target 
frames chosen for fusion from the video clip. 

The basic approach taken is to first divide the whole 
picture into elementary rectangular cells/blocks of the mesh, 
determine the direction of motion, calculate the drift of the 
pixel positions to determine the picture section to be fused 
and finally fuse the calculated pixel positions. 

The algorithm stated above reads in a base frame, 
BASE_IMG along with a target frame from the clip, 
NEXT_FRAME. BASEJMG is then divided into the cells of 
the mesh. The same coordinates of division are then followed 
to divide NEXT_FRAME image. Next a loop is started that 
repeats the further processing on each of these cells. This 
processing involves, dividing corresponding cells of both 
the images into nine equal parts. The partitions obtained are 
then compared for both the images using the correlation. 
The partition pair with minimum value for correlation indicates 
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Algorithm 3 : Image Fusion 



Given BASEJMG. NEXTJRAME 

1. Read in the actual reference image 
ORIGINAL J3ASE 

2. Divide BASEJMG into elementary rectangular 
mesh cells such 

that no part of the mesh lines is included 

3. Divide the NEXT_ERAME into cells with same 
coordinates as of 

the BASEJMAGE ceUs 

4. for all cells in the BASE_IMG, do 

5. BASE_CELL4- Current BASEJMG ceU 

6. NEXT JRAME_CELL4- Current 
NEXT_FRAME ceU 

7. Divide both cells in 9 equal partitions 

8. for all corresponding partition pairs in both cells, 
do 

9. 

C ORR_VAL[I ] ^- c orrelation(B ASE_PART : NEXT_PA 
RT) 

10. end for 

1 1 . DIREC TION<eMax{C ORR_VAL[ ]) 
12. 

H_D RIET<e Eind_horizontal_drift{NEXT_FRA^ffi_CE 

LL) 

13. 

V_DRIFT-e Emd_vertical_drift{NEXT_ERAN-DZ_CELL 

) 

14. for Drifts in both orientations, do 

15. From the edge of cell, move DRIFT pixels 
away along the 

length of edge 

16. Fuse this calculated section of selected pixels 
from the 

NEXTJRAME into BASEJMG 

17. end for 

18. end for 



the direction of the motion. After this, the Horizontal and 
vertical drifts of pixel positions from BASEJMG to 
NEXT_FRAME are calculated in accordance with the direction 
of the motion. With these steps all the necessary data for 
fusion is available. This data gives the section of the image 
NEXT_FRAME to be fused with the BASEJMG. Fusion is 
performed to fill up the mesh lines in BASEJMG with the 
information extracted from the NEXT JTIAME that gives the 
resultant image free of noise i.e. the mesh. 

VII. Results And Discussion 

This paper shows the practical implementation of the reg- 
istration process that the human brain does automatically 
when some one looks at an object through the mesh and by 
moving the head or the mesh, he is able to recognize the 
object completely. The results from the applied method have 
been quite encouraging. The extent of the distortion in the 
results is observed to be sufficiently low. The data for the 



application were taken under the assumptions that the cam- 
era motion involved only horizontal and vertical translations 
without any rotations, that the motion is confined to a single 
plane and that the lighting conditions and background view 
do not change considerably. Although the rotational vari- 
ances could be nullified by applying opposite rotational trans- 
formations on the extracted frames, the process was avoided 
in the samples for the sake of simplicity. 

VIII. Conclusion 

The method was tried on 200 images having wire mesh as 
on the fore ground. This method removed all of them. The 
method works very well till the background does not change 
but need modification for accommodating the change in the 
background. It has been realized that the background change 
due to shadow only disturbs the registration process. This 
can be overcome by re-registration process but it is time 
consuming. The application is new and the method is 
insensitive to any kind of skew occurring due to the 
movement of camera in unknown direction. 
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